By Steve Partlow, advisory software engineer, z/OS design/development
A challenging project is always more fun; and in that respect the project to exploit Flash Express for z/OS paging brought plenty of delight. I joined the Storage Management team in 2009 to participate in the design and begin writing code in support of Flash. For this project the team went from what had been 10 people to a total of 18. They ranged from long-time storage management experts, to those who had previously been a part of the team, to storage management novices, like myself.
The z/OS Storage Management components are like a well oiled machine that is tuned for what it does: managing 4K pages of data between real storage and auxiliary storage. So it was a challenge to not only add a completely new medium of storage but to go from an all 4K paging paradigm to one that could handle 1M large pages as well. Each step of the way we had to pay special attention to continue to efficiently manage mainly 4K pages while making storage management more versatile and flexible with 1M page support.
Because the project required so many different pieces from multiple z/OS components, one of the first difficulties was adding code to the system that could be tested before all dependencies were complete. This required a lot of scaffolding and test hooks which were later removed or disabled as each piece was completed. It was “throw away” code but allowed us to ensure a higher quality than waiting to start test until many thousands of lines of code could be run together.
I had never worked on a project with such a direct connection to new hardware and doing so brought its own challenges. We performed our early tests on simulation which mostly, but not always, behaves like the hardware. One major difference between simulation and the hardware is performance. Not only is hardware faster but it has different timing characteristics. After beginning tests with the real Flash Express hardware we then began to optimize our algorithms to improve performance on the new architecture.
Because z/OS Storage Management is the first exploiter of Flash, our team also became the gatekeeper of Flash for z/OS, managing what portions are already obtained and to whom. We laid the foundation for future enhancements by providing access to other exploiters so they will be able to access Flash without needing to know all the intricacies of the architecture.
Not only is z/OS Storage Management itself evolving, so is its development process. For this project we decided to dive into agile software development techniques. This wasn't easy with such core z/OS components but we were able to break down many functions to define user stories which could each be completed in four weeks or less. Using such small user stories felt foreign, even artificial in some respects, but it also forced us to have testable code faster and allowed us to find weak spots earlier in the development cycle.
This project presented many challenges which the team met with gusto. I'm proud to have been a part of the team and look forward to tackling even more enhancements in the coming years.
This blog post is part of a four part series on FlashExpress from the Flash development team. Check out other posts on the Mainframe Insights blog:
Steve Partlow is an advisory software engineer in z/OS. He has 12 years of experience in z/OS Storage Management (ASM, RSM, VSM) and Global Resource Serialization (GRS). He enjoys his two young boys, running, and other challenges.