>
Background
It's a 2D sandbox game set in a procedurally generated underground world. It shares mechanics with Minecraft, Terraria, and Stardew Valley, such as crafting, exploration, and combat [1].
It uses a top-down perspective with thematic influences from dungeon-crawler RPGs [2].
It is currently in early access. Our playtest focuses on solo play, but it also allows multiplayer.
Tools
Participants
Target audience
Adolescents and up. People who enjoy genres: “Sandbox”, “RPG” and “Action / Adventure”
Participants
All completed a screening survey.
The graphs show the data for the eight selected playtesters.
Participant data
Age range = 20 – 24
Mean age = 21.75
Genres liked by participants
Hours spent gaming per week
Gender - Participant demographics
Self identify as gamer
Potential playtest participants were screened using a survey due to the limited number of playtests we could perform in the time available. We sent this survey to university students as an opportunity sample due to their high availability for an in-person playtest on campus.
They were first asked if they had played Core Keeper before; none had, but they would have been rejected as we aim to evaluate the first-time user experience (FTUE).
We used Steam’s genre tags [3] and reviews in the game press [1] [2], to classify Core Keeper as “Sandbox”, “RPG”, and “Action / Adventure”. We assume the target audience will enjoy these genres; therefore, we asked participants to select their top three genres from a list of ten; participants who did not select at least one of Core Keeper’s genres were rejected (two out of ten respondents).
We also wanted a range of gaming experience levels across our participants. To ensure this, we asked participants if they self-identified as a “gamer”, the time spent playing video games per week, and the genres enjoyed. This was influenced by the social identity approach to classify gamers by Ćwil and Howe [5]. One participant did not describe themselves as a gamer and did not play video games consistently enough to quantify a weekly average playtime. We still selected the non-gamer for playtesting as they enjoyed two of the game’s main genres and we believed it was valuable to test the game with a very “casual” player.
Study Design
Goal: Evaluate the first-time user experience (FTUE).
Specifically, how players learn game mechanics if they are motivated during and after the gameplay to continue playing and their initial impressions.
Data collection methods
The participants were asked to think aloud during the playtesting session. This was chosen because we wanted to gauge their thought processes and reactions. Concurrent think-aloud was appropriate as sandbox gameplay does not have high cognitive demands, and the player largely directs the pace.
After the playtesting, we conducted a semi-structured interview. There was a range of questions to probe player’s recall and understanding of game elements and their motivations behind certain gameplay choices. There were also more open-ended questions for players to comment on their memorable experiences and feelings. This complemented the concurrent think-aloud by allowing us to dig deeper into why the players made certain choices rather than just reporting on their actions.
During the playtesting, we noted down our thoughts and direct observations. We also captured screen and voice recordings of the participant’s gameplay and interviews. The notes provided structure and useful context for what occurred in the session, and the recordings allowed us to get direct quotes, which facilitated thorough nonlinear data analysis.
Procedure
After analysing the responses from our screening survey, we selected eight participants. We arranged a suitable time and location for the session with the participant (either Piazza or the Library).
All playtests were conducted on the same version of the game played on the same laptop model with a mouse; this was a conscious choice to standardise our experimental conditions.
All participants completed the consent form, read the information sheet and were given an opportunity to ask questions. A team member then verbally briefed them on the game and method. Then, we set up the screen and audio recordings. A 25-minute timer was set for the session, and participants were asked to start their gameplay and think aloud.
During playtesting, the players were directly observed. The moderator occasionally prompted the participant to continue thinking aloud if needed. Additionally, if a player felt stuck, the moderator sometimes provided hints to facilitate progression in the game.
After the time was up, we stopped the screen recording and set up the voice recording for the interview. The moderator led a semi-structured interview with each participant. Based on their responses and the results of the gameplay, the moderator occasionally tailored and provided follow-up questions to get more in-depth data on specific aspects of the game, which may improve data quality. After the interview, we turned off the recording and concluded our playtesting session and debriefed the participant.
Playtesting setup
Analysis
We used inductive thematic analysis to analyse our concurrent think aloud and interview data. Thematic analysis is a qualitative analysis method for “identifying, analysing, and reporting patterns (themes) within the data” [6]. Our method followed the framework of Braun and Clarke’s (2006) [6].
We chose to code inductively as we determined there were no relevant priori codes in the literature because there were no similar studies on first-time player experiences of games of its genre. Since think-aloud and interviews varied significantly between participants, it allowed us to interpret and classify the themes of the participants as a whole. With this method, we could comprehensively identify and organise our findings based on the themes generated from the data.
We used screen/audio recordings and our notes to familiarise ourselves with the data before performing open coding in a shared spreadsheet. From these codes, we generated themes for each participant. We compared and grouped codes on sticky notes to discover the most prevalent themes.
We identified 55 codes and categorised them into seven themes. The most numerous and relevant themes to FTUE are as follows:
1) Poor information for key mechanics. 2) Undefined end goal. 3) Health and food top bar not noticed. 4) Lack of narrative engagement. 5) Map UI unclear. 6) Audio/Visual Feedback from Actions. 7) Standard input controls.
Our process of generating themes
Findings
Audio/Visual Feedback from Actions
The vast majority of players felt the game offered an engaging level of audio/visual feedback when their character performed an action. Visual cues such as the block-breaking animation did not just provide utility but also provided playfulness. P4 = “There’s this thing it does when you mine, where the wall will bounce, which I think is really cute. Just small things like that is really enjoyable when you’re expecting feedback from an action.”
Many players appreciated the game’s comprehensive range of sound cues when interacting with the environment, such as breaking or placing different materials and attacking enemies. P3 = “When you did stuff, there was always some sort of sound cue [...] Whether it was mining, there was like the block breaking, and then there was like a ‘tinkely’ noise.” P5 = “you could hear when you hit enemies, which I thought was good.”
Participants who were initially struggling with some of the game mechanics relied heavily upon this audio/visual feedback to update and reinforce their mental models. P6 = “There was a good level of feedback [...] when I did something the feedback reinforced what I had done and slowly helped me figure out what the hell I was doing.”
Standardised input controls-control menu
All players quickly picked up basic controls, allowing them to move and easily interact with the world. Players with a high level of gaming experience felt they were easy to figure out, and the game’s current introduction was effective. P1 = “I don’t think there’s a need for a tutorial; it’s quite intuitive.”
Players with a low level of gaming experience, such as P6, a self-described non-gamer, also recognised the basic controls relatively quickly and were almost fluent by the end of the session. P6= “The arrow keys came straight away [...] the one thing that was interesting was initially I was spam clicking for walls and then realised you can press and hold instead.”
P6 was the only participant who used the arrow keys; the rest used WASD, and the fact the game supports both allowed for a seamless first-time user experience.
Poor information for key mechanics
Negative issue:
Many players felt the game did not provide them with sufficient information for utility blocks/tools. P5 = “you kinda just gotta figure it out as you go along, which isn’t the most helpful - at least not to begin with.” For example, all three players who used the cooking pot assumed the bottom slot was for fuel. P5 = “Does it need fuel[...], does it need water maybe, did it say?” They did not realise they had to combine two food items to cook a meal. Hunger is a core game mechanic. If inexperienced players are blocked from cooking, they starve. All players who tried farming did not know if they needed to water the seeds for them to grow as the description does not say it needs to be watered, and some at first thought a spade was used to till the ground, not a hoe. P7 = “I wonder if I need a water source or anything, or I just need to place the seed down and hope it will just grow.”
Poor information for key mechanics
Suggestion:
Tooltips on key items and utility blocks could be updated to provide more logistical information concerning their game mechanics. They could use simple plain language and avoid euphemisms which may be misinterpreted by inexperienced players.
Undefined Goals and Objectives
Negative issue:
The majority of our participants reported a lack of objectives, making them feel confused or unsure how to progress. P5 = "Not sure what I am meant to be doing, there isn't any goals , and if there is one, how do I get there."
Some felt the lack of immediate objectives made the overall aim of the game seem unclear. P1 = "The game just puts you here and didn’t tell anything [about what] you should do next or what’s your final targets. I’ve played some other 2D mining games but it will have a mission [...] on the top."
A few participants were able to infer from the start area surroundings that the end goal is to power up the core with crystals to escape the mines, but many were unsure how to make meaningful progress towards it beyond exploring.
Suggestion:
While autonomy in sandbox games are important for motivation, some guidance would be appreciated by the player. Similar games like Stardew Valley [8] have missions for the player to complete which facilitates learning of game mechanics and game progress.
The game could have missions to provide the player with objectives. For example, fetch quests to introduce players to game resources and tools. The overarching objective of escaping the mine could also be introduced this way with some indication that it is a multi-step process and could reveal more as player progresses.
Health & Food top bar not noticeable
Negative issue:
Many players overlooked the game’s health and food bars. They were unlabelled and did not display a numerical value. Also participants were focused on their character in the centre of the screen which sometimes resulted in frustrating deaths or repeated prompts that they were starving. P3 = “There aren’t too many tips on the top bar, I died a lot because I didn’t notice it.” P7 = “I wasn't paying attention […] to the food meter so it seems I'm starving.”
Suggestion:
Providing labels and numerical values alongside the bars would ensure players could understand their status at a glance. “The Legend of Zelda: Breath of the Wild” uses both visual bars and numerical values to display Link's health and stamina. The bars change colour and flash when they are low, drawing the player's attention to their status [7].
Only using a health bar to convey character health has been criticised in game design literature as, other than when empty, often their state does not have an effect on the game making it seem
“flat” [10]. While exploring the world with low health there could be an atmospheric change indicating the urgent nature of the character condition. For example, the screen flashing red or
a change in music.
Unclear Map UI
Negative issue:
A lot of players did not find the map feature helpful, especially at the beginning of the game when players had only explored a little. During that point in gameplay P6 = "I am sure I am missing something obvious in terms of getting out of the small area I am currently in”, however when looking at the map said “[This is] some sort of map but I don't understand how to use it at the minute". A lack of appropriate labelling in the map UI confused players as it was unclear what was being represented on the map or how it should be used.
As players explored further into the game, the map gained more utility as it tracked the player’s current location and areas they had explored P5 = “The map is sort of useful, I mean I see clearly where I am at least.”
The game also allows users to put custom pins on the map to aid with navigation but none of our participants figured out this mechanic. P5 = "On the map, it doesn't say what anything of these [the pins] are here either."
Suggestion:
Rather than having a largely solid black map at the beginning, unfiled outlines could indicate the presence of nearby areas of interest, similar to the map mechanic in Assassin's Creed Odyssey [11]. A legend could be added so players can easily identify their discoveries. Instructions could also be added to the pin section at the bottom left of the map screen.
Lack of narrative engagement
Negative issue:
Most players do not find the narrative salient. Half of the participants felt the story could have been more engaging. This included P2, who described it as “[To be] brutally honest, pointless”, and P3, who skipped the opening cutscene entirely. The other half of participants generally had trouble recalling it, some describing it as generic, as they felt the narrative was not supported by the gameplay. P3 = “The first bit could have been cool, [...] but then there was like no further story element afterwards there was no explanation of [...] what are you supposed to be doing”. While some players felt they did not engage with the narrative, we found it did not affect their overall engagement with the game.
Suggestion:
To avoid it feeling unimpactful, the amount of text in the opening cutscene should be minimised to take a “show, don’t tell” approach, as video games are a visual storytelling medium [9]. To keep players engaged, the sequence could also utilise stylised animations rather than static images. An alternative approach would be to drop the initial passive cutscene and replicate it with active gameplay, which would be more engaging. As Skolnick writes, “in games the old Hollywood axiom can be amended to, “do, don’t show” [9].
Strengths
Conducting a screening questionnaire helped curate our playtesting sample to more accurately represent our target audience than if we just recruited from our opportunity sample.
The thematic analysis approach was useful to help us make sense of our findings. Many of our codes shared the same causal mechanism, and by grouping them, we could better identify issues and suggest solutions.
During the playtests, we wanted to avoid crowding the participants when they were playing; however, when making notes, the screen could sometimes be unclear from a distance. The gameplay recordings were very useful in complementing these notes when analysing the gameplay.
Semi-structured interviews were beneficial as we had many consistent questions between participants that we could compare and contrast. Still, we could also ask specific questions drawn up from what we saw during the playtest and ask follow-up questions to probe further into their experiences.
Limitations
Using inductive thematic analysis for analysing think-aloud and interview data took significant time to develop codes and themes properly. A good portion of our discussions were taken up by developing themes that would be inappropriate for our research aims and so had to be iteratively redefined. If this project were done in industry, time scales would be tighter, so it would be an inappropriate method.
We coded the games and interviews manually in a spreadsheet. It was especially time-consuming to transcribe what the participants said. Keeping track of the codes was challenging as we had to keep switching between different cells/sheets. In the future we would consider using dedicated qualitative analysis software such as nVIVO.
We recognised that adolescents were part of the target audience of Core Keeper; however, due to the nature of the assessment, participants were required to be over 18. Our screening questionnaire was also distributed among people known to the researchers so the participants may differ from the broader range of target users in terms of age and interests.
The sample size of our screening survey was relatively small (n=10), and therefore, we only had one participant who liked all three of Core Keeper’s main genres. A larger sample size for the screening would have allowed us to recurit participants who are even more aligned with the target audience for the limited playtesting slots.
Reference
[1] N. Clark, (2022). “Core Keeper makes the sandbox survival sim formula feel new”. [Online] Polygon. Available at: https://www.polygon.com/gaming/22991657/core-keeper-survival-sim-early-review. [Accessed 25 Apr. 2024].
[2] R. Lane, (2022). “‘Core Keeper’ is an atmospheric survival sim that reminds me of ‘Dungeon Keeper’”. [Online] NME. Available at: https://www.nme.com/features/core-keeper-is-an-atmospheric-survival-sim-that-reminds-me-of-dungeon-keeper-3190614 [Accessed 29 Apr. 2024].
[3] Pugstorm, Core Keeper. (2022). Fireshine Games. Accessed: 26 Apr. 2024. [Video game] Available: https://fireshinegames.co.uk/games/core-keeper/
[4] “Core Keeper,” Nintendo. https://www.nintendo.com/en-gb/Games/Nintendo-Switch-download-software/Core-Keeper-2480263.html [Accessed 29 Apr. 2024]
[5] M. Ćwil, W.T. Howe, 2020. Cross-Cultural Analysis of Gamer Identity: A Comparison of the United States and Poland. Simulation & Gaming 51, 785–801. https://doi.org/10.1177/1046878120945735
[6] V. Braun, V. Clarke, (2006). Using thematic analysis in psychology. Qualitative Research in Psychology 3, 77–101. https://doi.org/10.1191/1478088706qp063oa
[7] Nintendo EPD, The Legend of Zelda: Breath of the Wild. (2017). Nintendo. Accessed: 30 Apr. 2024. [Video game] Available: https://www.nintendo.com/en-gb/Games/Nintendo-Switch-games/The-Legend-of-Zelda-Breath-of-the-Wild-1173609.html
[8] ConcernedApe, Stardew Valley. (2016). ConcernedApe. Accessed: 30 Apr. 2024. [Video game]. Available: https://www.stardewvalley.net/
[9] S. Evan, “Part 1: 5 Exposition” in Video Game Storytelling: What Every Developer Needs to Know About Narrative Techniques. New York: Watson-Guptill, 2014, pp. 56–58.
[10] K. Burgun, “Chapter 5. Through the Lens: Video Games” in Game Design Theory: A New Philosophy for Understanding Games. Boca Raton, Florida: CRC Press, 2013, pp. 119.
[11] Ubisoft Quebec, Assassin’s Creed: Odyssey. (2018). Ubisoft. Accessed: 1 May 2024. [Video game]. Available: https://www.ubisoft.com/en-gb/game/assassins-creed/odyssey
Thank you for your attention.
