Introduction
Testing of the Twitch Plug-in is divided into four parts. First, a requirements testing checklist which is detailed in this document. Second, a longevity stress test where the plug-in is run for a long period of time. Third, a ‘developer-local’ dry run where the plug-in is used with live data but not displayed to viewers. Lastly, a live test that will occur prior to the official charity stream date.
Requirements Testing Checklist
This checklist defines tests to verify that all user and technical requirements have been met. The test cases listed per requirement were created as a collaboration between myself and the client. Test plans were run periodically throughout the project, but a full pass was run during the first hardening sprint and again at the end of the project. In addition to running the checklist on my own, I sat down with the client and ran through the user requirements test list with them to confirm everything was working as they expected.
A few requirements changed during the project (reflected in the updated requirements document). These adjustments have also been noted in the test plan.
User Requirements
| ReqId | Requirements | Test Actions | Status | Notes |
| 1 | Stream countdown timer | |||
| 1.1 | On-screen widget that is hideable by viewers | N/A | N/A | This is not possible without a Twitch-integrated extension, which was determined not to be the ideal approach for this project |
| 1.2 | Displays Hours, minutes, seconds left in stream | – Verify time is shown in hours, minutes, seconds – Verify that when timer is counting down, it counts down in seconds – Verify that minutes and hours are decremented at appropriate times as timer updates |
Success | |
| 1.3 | Updatable in real-time (see donation timer impacts) | – Verify that timer is updated for each donation type: – Twitch subscription – Twitch follow – Twitch cheer (bits) – Extra-life donation – Verify that in each of the above cases, the timer is updated by the amount of time the user has input into the config page |
Success | |
| 1.4 | Timer is pauseable by streamer | – Verify that the timer pauses and starts when the pause and start timer buttons are pressed on the config page – Verify that pressing the pause button while paused, or the start button while the timer is running doesn’t affect the timer |
Success | |
| 2 | Donation timer impacts | |||
| 2.1 | Donations to twitch in the form of subscriptions, bits, follows increase remaining time of stream | – Verify that timer is updated for each donation type: – Twitch subscription – Twitch follow – Twitch cheer (bits) – Verify that in each of the above cases, the timer is updated by the amount of time the user has input into the config page |
Success | |
| 2.2 | Donations to extra-life site increase remaining time of stream | – Verify that timer is updated for each donation type: – Extra-life donation – Verify that in each of the above cases, the timer is updated by the amount of time the user has input into the config page |
Success | |
| 2.3 | Timer is updated within 1 minute of donation being processed | – Verify that the time between a donation being triggered, being processed by the server, and reflected in timer is less than 1 minute | Success | Testing showed that the update occurs between 1 – 2 seconds in the case of Twitch events, and up to 20 seconds in the case of Extra-Life events |
| 3 | Donations config | |||
| 3.1 | Streamer can access configuration window outside of stream | – Verify that config page can be accessed outside of OBS within browser of choice | Success | |
| 3.2 | Streamer can set stream time increases by donation type – Dollars for extra-life donations – Per Subscription – Per bit – Per follow |
– Verify config page form contains fields for all donation types: – Twitch subscription – Twitch follow – Twitch cheer (bits) – Extra-life donation – Verify that these numbers can be saved to config and are reflected in the timer when new donations occur |
Success | |
| 3.3 | Donation configs are entered in minutes seconds | – Verify donation form fields treat inputs as seconds and are reflected as such when donations occur | Success | After testing the interface with the user, it was determined that these numbers should be in seconds instead of minutes |
| 3.4 | Streamer can enter time remaining manually | – Verify there are fields for hours, minutes, and seconds on the config page form – Verify that these fields are reflected in the timer after the user clicks ‘update config’ while timer is paused and while timer is running |
Success | |
| 4 | Donation stats tracker | |||
| 4.1 | Displays as a dropdown within timer widget (hidden by default) | N/A | N/A | This is not possible without a Twitch-integrated extension, which was determined not to be the ideal approach for this project |
| 4.2 | Viewers can press show/hide button within widget to hide stats while still displaying timer | N/A | N/A | This is not possible without a Twitch-integrated extension, which was determined not to be the ideal approach for this project |
| 4.3 | Displays the most recent donation, who donated, the amount (and type) of donation, and time added | – Verify most recent donation is reflected for all donation types – Verify the amounts and types reflect the donations that occur – Verify that the list shows the top 3 donors – Verify the top donor list updates as new donations come in – List should reorganize properly based on where the donation exists within the leaderboard – List shouldn’t change if the recent donation is less than the top 3 |
Success | |
| 4.4 | Displays the top 3 donations since streamer-defined date from the streams where this plug-in was used | – Verify that the top 3 donations from all active streams that this plug-in is used are shown | Success | After testing with the user, they decided they would like the top 3 donations to always persist for any streams that this plug-in is run, regardless of date. |
| 5 | Donation Announcements | |||
| 5.1 | Post chat message when donations occur (who, type, amount, time added) | – Verify a message is posted to Twitch chat using the streamer’s username – Verify message includes: – Who donated – Donation amount – Donation type – Time added to stream |
Success | |
| 5.2 | Streamer can configure custom message to follow the above information in chat | – Verify message that user’s custom message on the config page is displayed in chat message following the donation information. | Success | |
| 6 | Basic donation effects | |||
| 6.1 | Display an on-screen video using ‘Pingu Mozart meme’ – Displays amount donated in ‘happy’ part of video – Displays amount of time added in ‘terror’ part of video |
– Verify video isn’t shown while notification is inactive – Verify video plays from start when a new notification occurs – Verify video disappears after notification ends – Verify that donation text displays in first part of video – Verify that time added text dispalys in second part of video |
Success | |
| 7 | User Donation Debug Tools | This was an additional request from the user after initial testing. The purpose of this feature is to test how everything interacts with the streamer’s stream layout prior to going live. | ||
| 7.1 | User can trigger ‘fake’ donations to see notification and verify timer is working correctly | – Verify config page has fields and buttons that allow user to trigger fake donations | Success | |
| 7.2 | Donations should be of any valid type with customizable amounts in the cases of cheers (bits) and extra-life donations (dollars) | – Verify there are amount fields for bits and extra-life donations and these numbers are reflected when donations are triggered – Verify that there is a button for follows and subscriptions without amount fields |
Success | |
| 7.3 | User should be able to enter a fake donor name | – Verify donor field name exists and that this is represented in any fake donations that trigger | Success |
Technical Requirements
| ReqId | Requirements | Test Actions | Status | Notes |
| 1 | Config Page | |||
| 1.1 | Save/Load | – Validate donationConfigStore.json and timerConfigStore.json are updated when ‘update config’ button is pressed – donationConfigStore.json should contain all time values set per donation types – timerConfigStore.json should contain the remaining stream time |
Success | |
| 1.2 | Event subscriptions | – Validate config page receives timer config and donation config update messages – this should be reflected after initially launching the page as all existing values should populate |
Success | |
| 1.3 | Visuals | – Page is not viewer-facing, so this should value functionality over aesthetics – Validate simple page with clear form and buttons |
Success | |
| 1.4 | oAuth redirect | – Page should contain a link for user to validate with Twitch – This should redirect to twitch oauth page if user hasn’t already validated – After authenticating, should redirect back to config page |
Success | |
| 2 | Timer Page | |||
| 2.1 | Save/Load | – Validate timerConfigStore.json is updated as timer ticks | Success | |
| 2.2 | Event subscriptions | – Validate that timer receives pause/start events | Success | |
| 2.3 | Visuals | – Validate that timer uses official Extra-Life branding and design is signed off by user | Success | |
| 3 | Stats Page | |||
| 3.1 | Save/Load | – Validate donationStatsStore.json is updated with recent donor and top 3 donors as donations occur | Success | |
| 3.2 | Event subscriptions | – Validate stats page receives all donation stats update events | Success | |
| 3.3 | Visuals | – Validate that stats page uses official Extra-Life branding and design is signed off by user | Success | |
| 4 | Notification Page | |||
| 4.1 | Save/Load | N/A | N/A | Notifications require no save/load functionality |
| 4.4 | Event subscriptions | – Validate notification page receives all donation events | Success | |
| 4.5 | Visuals | – Validate that stats page uses official Extra-Life branding and design is signed off by user | Success | |
| 5 | Server | |||
| 5.1 | Twitch Event Sub connections | – Validate app secret is input in user config file and not stored in source code – Validate server uses proper Twitch-approved oauth techniques to authenticate with event sub server – Validate that connection is successfully established between event sub and server. – Validate that server connection is kept alive and maintained for at least 24 hours – Validate that server handles reconnection requests – Validate that follow, subscribe, subscription.message, and cheer events are all received from event sub server |
Success | A 48-hour test was run, connection was maintained during this time. Reconnection and re-auth attempts were successful during this time. |
| 5.2 | Twitch Chat connections | – Validate broadcaster information is contained in input file and not stored in source code – Validate server uses proper Twitch-approved oauth techniques to authenticate with chat server – Validate that connection is successfully established between chat server and server. – Validate that server connection is kept alive and maintained for at least 24 hours – Validate that server handles reconnection requests |
Success | A 48-hour test was run, connection was maintained during this time. Reconnection and re-auth attempts were successful during this time. |
| 5.3 | Extra-life API calls | – Validate that extra-life API is polled every 5 seconds – Validate that donation information is retrieved and parsed correctly, identifying new extra-life donations |
Success | |
| 5.4 | Subscription maintenance and notifications | – Validate server handles incoming notification subscription requests from client-side pages – Validate server stores these subscriptions – Validate subscriptions are removed when a client socket connection is closed – Validate that current known information is broadcasted to new subscribers – Validate Notifications are broadcasted to all applicable subscribing clients |
Success | |
| 5.5 | Client connection management | – Validate that clients are able to establish a socket connection with server within 1 second of launching – Validate a unique UUID is assigned to each client socket connection |
Success | |
| 5.6 | Save/Load | – Validate that server is able to write to and retrieve data from all storage files – donationConfigStore.json – timerConfigStore.json – userConfig.json – Validate that data is corrected stringified and parsed by JSON upon storage and retrieval – Validate that server precaches current data stored in JSON files (if existing) upon server launch |
Success | |
| 5.7 | Message handling | – Validate that server handles the following client message types: – updateTimerConfig – postTimerUpdate – updateDonationConfig – subscribe – startTimer, pauseTimer – twitchSub, twitchFollow, twitchCheer, elDonation – requestTwitchAuthUrl – authenticateTwitch |
Success | |
| 5.8 | Debug options for Twitch | – Validate that a debug flag is available within serverTwitchHelpers.js that will connect to Twitch CLI test servers instead of prod servers – Validate that messages can be properly sent and received from Twitch CLI test server |
Success | |
| 6 | Technology – General | |||
| 6.1 | Must abide by allowed Twitch technologies – HTML, CSS, JS for extension code – Provided twitch API |
N/A | N/A | This is no longer a requirement, as the client and I opted for a local plug-in solution instead of an integrated extension |
| 6.2 | Must abide by API limits put in place by Extra-Life and Twitch (ie. Num calls/minute) | – Validate we only call the extra-life API a maximum of once every 15 seconds, with an additional call during that time if changes are detected – Validate there is a maximum of 3 socket connections with the Twitch event sub server, with a maximum of 100 subscriptions per socket |
Success | These limits are based on the DonorDrive API guidelines for Extra-Life (https://github.com/DonorDrive/PublicAPI) and the Twitch Event Sub guidelines (https://dev.twitch.tv/docs/eventsub/handling-websocket-events/#subscription-limits) |
| 6.3 | Must abide by Twitch extension policies listed here https://dev.twitch.tv/docs/extensions/guidelines-and-policies/ | N/A | N/A | This is no longer a requirement, as the client and I opted for a local plug-in solution instead of an integrated extension |
| 7 | Security | |||
| 7.1 | Must not expose any sensitive information of streamer within extensions (IP address, name, etc.) | – Validate that app client-id and secret are stored in a user-input file instead of in code – Validate that no broadcaster or streamer information is hardcoded |
Success |
Longevity Stress Test
I ran the server along with the front-end pages for 48-hours (a period typically longer than the charity stream). During this time, I verified that all oAuth token refresh requests, server connections, and APIs remained stable and connected. This test was successful.
Developer-local Dry Run
The purpose of this test was to run the plug-in within a production environment without exposing the plug-in functionality to viewers. To do this, the streamer authenticated the plug-in with their Twitch account on a development PC. They then streamed from their normal PC while I monitored the development PC, verifying that the server and front-end pages all ran correctly during this stream. We ran the stream for about 5 hours (enough time to allow the oAuth token to need to be refreshed). This test succeeded.
Live Test
This test has yet to happen but will involve running the stream with the plug-in enabled and viewer-facing prior to the actual charity stream. The streamer preferred to delay this test to a more strategic time for donations. Given the delay we discussed the risks of deferring this test and determined that, after the success of the other tests, this would be low risk and was acceptable to sign off on the project without this being complete.