This week, we interviewed Jedrzej Kaminski, a Product Manager at EyeEm. Jedrzej shared his expertise in A/B Testing and data analytics to discuss measuring success when conducting research. He also answered burning questions on technologies he uses on a daily basis, his experience at EyeEm, and how to make different transitions into Product.
What’s your background, and what are you working on?
I have a master’s degree in psychology, but was always interested in technology – even my final thesis was about an online adaptation of pen-and-paper test to measure creativity, so I spend half the time doing research and half coding. I’m working right now as a Product Manager for EyeEm – an AI driven Marketplace for photos delivered by a vibrant and talented community.
Can you describe your current role and how you got started in product management?
My career in IT started in Quality Assurance. While performing other responsibilities like manual and automated testing, I’ve found that talking to our community, participating in the user research and crunching the analytics numbers are quite enjoyable. They were also very close to the user-centric mindset I gained when trying to find bugs before our users. I wanted to try the Product role and EyeEm gave me an opportunity to do that. Now I’m a part of two cross-functional teams focused on driving business KPIs.
Could you tell us about your day to day? What technologies and methods do you use on a daily basis?
My day to day activities heavily depend on where are we, as a team, in the project development cycle.
During product discovery it’s more about analyzing the data, user behavior, quantitative and qualitative, to find the problems, frictions and struggles we can help them with. Formulating those problems is crucial, because then we know that the team work can be most impactful. Amplitude, Google Analytics, Tableau or just doing SQL queries – those are some of the tools that are helping a lot with quantitative analysis. You can also use Typeform or even Google form for more open, qualitative responses, but live user interviews are always a goldmine of insights. Obviously, nowadays, when we tackle the COVID-19 issue, we are doing video interviews instead.
After that it’s all about planning and ruthless prioritisation and looking for the smallest possible implementation of the feature that you have in mind, that can give us data to confirm that we are going in the right direction. You don’t want to waste resources on something that will not bring any value – to the user or the company. At the same time, while everyone is trying their best to mitigate that risk, be prepared for a situation when most of the feature ideas will have little to no impact on the metrics. This means that you need to try a lot, learn from the failures, and improve over time. Treat features as tests, with hypothesis and ruthless data validation.
Then we can focus on crystalizing the concept and execution. During this phase the workload consists of, but is not limited to; managing stakeholders, creating and maintaining tickets, design reviews, UI tests, organising documentation, preparing copy or translations, flowcharts and maintaining information flow in case of dependencies between different teams. Figma is one tool that really grew on me – as the place for design specification, but also a channel for communication between the developers and designers. At the same time it can be easily used to visualise user flows or present early wireframes. Other than that – Confluence, Asana and Jira for documentation and keeping the project running with an overview on the finished user stories.
And again – since you want to validate the impact of the feature, not just ship another thing, treat the release as a start of the test. Some tools can help with that – from in-house baked AB testing frameworks to using 3rd party content management solutions to drive your tests. We have a very nice setup with the mobile team, where we can, with some parts of the app, use Firebase, Contentful and Amplitude to randomize testing variants and create different groups with limited need for dev resources.
What are the biggest challenges you’ve faced and obstacles you’ve overcome as a PM?
Starting with a personal one – I’m a fairly introverted person and public speaking can be quite a stressful experience for me. At the same time driving product requires not only moderating smaller and bigger meetings with stakeholders, but also more public presentations about team vision, failures and successes to people inside and outside the company. The solution for being stressed and overwhelmed was quite easy – do the public presentations very often, push yourself out of your comfort zone, until it’s no longer a problem.
The biggest product challenge that I remember was my first bigger feature release, that was also changing the major functionality of our mobile app. Happy with the user test validations results during the planning phase, after the release happened, we were looking forward to measure the positive impact. The impact was there, sure, but more on the negative side. While current active users seemed unaffected, it was clear that new users are struggling – engagement and retention metrics were going down significantly.
The good thing was that we had built this new feature having A/B testing framework in mind. It took us about a week to analyze the tracking data and affected users flows, provide hypotheses on how to fix the unwanted behavior and design and ship experiments to gather hard data. In two weeks after the release we could clearly see that some variants were actually outperforming the old implementation in a significant way – so we went with them. To me the real success of this release was not only that we’ve delivered new functionality and value to the user, but that we could identify so quickly that something is going wrong and react swiftly in an impactful way.
What are your favorite ways to learn more about your users?
When it comes to qualitative research I really like going as close as I can to the concierge test – so basically do the user task for them, with prior training when needed, in the natural customer environment. When nicely structured it has most of the pros of normal user interviews but exceeds when it comes to learnings about the everyday user process problems. It adjusts your perspective and adds, in some cases extremely valuable, context of the user environment. I would love to do it more often. I’ve done it previously with photographers and loved the results.
Other than that – crunching the behavioral data numbers and conversion flow analysis. Properly tracked user behavior can make all of the user problems more visible and the solutions more achievable.
How often do you run meetings? How do you run it? Does your team run standups?
As a team we’ve embraced and found value in a couple of agile ceremonies. Due to COVID-19 it was also crucial to migrate all of those meetings online, so that we can continue to keep the level of productivity in the times of working from home.
Daily standup with tickets printed and moved on a whiteboard is now a Slack channel with Olaph bot collecting the standup answers. One upside is that it keeps the discussions focused, since we are using the threads on singular Slack messages – the team is not distracted and seems to like this solution.
Every week we are trying to groom our backlog – it’s the perfect way to keep it clean, estimated, and talk about new ideas that we think can be more impactful than projects on our current roadmap. New gen Jira along with Zoom are helping to make that meeting possible and efficient.
Every week we are hosting a User Insights session, when the data analyst or product people are presenting insights from the recent user interviews, behavioral data analysis or A/B test results. We want to be as data-informed as possible and utilize the whole team when it comes to looking for more detailed problems and solutions. Those User Insights sessions are designed to keep the information flow as wide as possible.
Every 2 weeks, when the sprint is over, we also run a retrospective. It’s very important to gather feedback and react, when something is going wrong – retrospectives are the team’s emotional buffer that generates tangible actionables to fix the behavior we want to mitigate. Our last retrospective was handled in Zoom, the “whiteboard” for the team to put the retrospective notes was done in Miro. To some extent it worked well, but next time we want to try using Figma instead.
Every 2 weeks we also have the sprint planning session. With the backlog groomed it should be a relatively quick meeting – we discuss the current roadmap, scope and the main goal of the sprint. We start the sprint as soon as the team is happy with the constellation of tickets.
Does your team use OKRs (objectives and key results)?
Our company moved to an OKR driven process about a year ago. We were doing them quarterly, now we are trying to set them per trimester. I believe that, if thought through, they are a very good tool to focus on important initiatives and can be used as a shield to protect you from distractions and team resource mishandling. Especially if you have many cross-functional initiatives. But I think the most important thing that OKRs are helping to popularize, is the idea that the team should not focus on shipping things, but shipping impact. The key result is measurable, so if the team member, mid-quarter, has an idea that everyone thinks is more impactful and easy to do than the current roadmap, OKR enables us, or makes it easier, to just adjust the roadmap.
How deeply should product people know about marketing strategy, UX design and coding?
I feel that when it comes to Product Management and a needed skill there is only one answer: “It depends, but for sure it will not hurt to know something about it”. Marketing strategy helps to push for better naming, copy, tackling user and business needs, release plans and so on. UX knowledge enables you to have more valuable conversations with designers, produce more focused wireframes, conduct better user interviews. Coding skills will for example help you understand the complexity of development tasks, technical limitations and opportunities, help you to analyze the data, along with more valuable conversations – this time with the devs. All of those skills will help you to create more precise and understandable user stories, conduct more focused and valuable meetings, and set more impactful strategies. All of those skills will help you to be a more valuable part of the team and a better Product Manager.
Are they absolutely and crucially needed? Not really. But just the fact that you are asking this question means, to some extent and expertise, that they are more than welcome.
Do you have coding skills?
As I said I was always interested in technology and coding as a way to interact with it. I Coded my final thesis during psychology studies, made two Android apps afterwards to check how hard it is to push something on my phone and to finish post-graduate studies. I then continued with working on automated tests and tasks. All this helped me to better understand my market domain, opportunities and the team needs. I’m not a developer, but I have enough knowledge to appreciate their work.
What skills do you think would be most valuable to learn and prioritize for an aspiring PM with no technical background?
I think that prioritization is one of the key skills and struggles nowadays – managing stakeholders, input from many sources, keeping the vision clear, team focused and inspired.
As the second one I would nominate the skill to conduct research – qualitative, quantitative and being able to analyze the data, find the patterns, identify the problems for the team to take care.
Then it’s time to communicate – within and outside the team, in tickets, documents, product demos or stakeholders updates. You need to be clear, efficient and convincing.
Focus is an OKRs platform that increases team alignment and performance. You can try Focus for using OKRs, running daily standups and weekly retrospectives. This mix of strategy and tactics allows to align the team each day and keeps focus on what really matters.