Stata Conference 2019

Barrel-aged software development:

brewscheme as a four-year-old

Billy Buchanan, Ph.D.
Director, Office of Grants, Research, Accountability, & Data
Fayette County Public Schools

https://wbuchanan.github.io/stataConference2019

The term "software development" implies some type of change over time. While Stata goes through extraordinary steps to support backward compatibility, user-contributors may not always see a need to continue developing programs shared with the community.

How do you know if or when you should add additional programs or functionality to an existing package? Is it easy and practical to extend existing Stata code, or is it easier to refactor everything from the ground up? What can you do to make it easier to extend existing code? While brewscheme may have started as a relatively simple package with a couple of commands and limited functionality, in the four years since it was introduced, it has grown into a multifunctional library of tools to make it easier to create customized visualizations in Stata while being mindful of color sight impairments. I will share my experience, what I have learned, and strategies related to how I dealt with these questions in the context of the development of the brewscheme package. I will also show what the additional features do that the original brewscheme did not do.

`brewscheme` a historical perspective

'The code you write is called a system when there is more than one related function that users will call.' (Gould, 2018; pp. 287)
Right out of the gates, brewscheme was off to a suboptimal start.
Instead of decoupling of concerns I tried to jam it all in there.
This was also compounded by hardcoding so much of the data used to construct the schemes.
Decoupling the concerns (e.g., reading vs writing) meant I needed either a public (new ado) or private (subroutine) function to handle the reading/importing.
brewcolors handled adding new named color styles to Stata.
brewextra was added to extend the palettes available from brewscheme.
brewdb was eventually create to handle building the database for the color palettes.
The Mata recycle function was inspired by the way R handles arrays (or vectors in their parlance) of different lengths.

dirfile gets used to construct/rebuild directories along the ADOPATH to store the data used by brewscheme/brewtheme.
brewterpolate provides a mechanism to identify colors with a specified number of points interpolated between a starting/ending color.
hextorgb was added to make it easier to work with hexdecimal colors.
brewviewer a previewer for color palettes
brewscheme was already verbose enough without adding all the other scheme elements to the program.
It was also clear that brewscheme had addressed one specific part of the problem space.
brewterpolate gets developed to start addressing the other part of the problem space.
This is also a great example of where Bill Gould's Mata book being published a few years earlier would have made my life much easier.
A day after the first draft of brewterpolate is committed I rewrite just about the entire thing with a completely different architecture.
Had I started writing all of this in Mata initially, it would have been much easier to extend things to handle the brewtheme case.
Writing things in Mata would have also made it easier to add additional color palettes.

Initially brewscheme only generated a single scheme file.
Due to some of the changes, additions, and updates made in this section, it became possible to make brewscheme generate schemes to simulate the user specified palette under different forms of color sight impairments.
The major contributor to this started as colorblind.mata which was subsequently refactored to become libbrewscheme.
Although some community contributors choose not to release Mata source, I took a different approach.
filesys uses Java to identify file system properties (e.g., file modified dates). This is used by brewlibcheck to detect whether or not the Mata library needs to be updated for the user automatically.
Unlike some of the other programs referenced here, brewtheme's help, filesys, and brewterpolate are all maintained in separate repositories and bundled with the system/toolkit together.
The help file for brewtheme may actually be one of the more important community contributions for documentation purposes (assuming nothing much has changed with the Scheme files).
Finding help entries related to scheme file properties is not easy by any stretch of the imagination.
In constructing the helpfile for brewtheme I quite literally searched for help file entries for every single property that can be set by brewtheme. When no help documentation could be located, I added references to which line in the s2color scheme the property could be found.
However, I view brewproof as the most important contribution from an applied perspective.
While it is great to be able to generate scheme files that simulate color sight impairments, it is better to be able to proof your visualizations before publication to ensure that the visualizations are accessible to all.
The most challenging issue to deal with when debugging the schemes was definitely the handling for contour plots. I remember this taking much longer than the 2-3 days between the GitHub issue and commit resolving it.
One of the challenges with developing a toolkit/system of commands is the API design itself. In this case, there were different options that all essentially meant the same thing (e.g., replace and rebuild). This is when it finally hit me and I fixed it.
brewcbsim this is a simple program that allows users to see how a set of colors might be perceived by individuals with different forms of color sight deficiencies.
brewtransform gets used internally when updated the database of palettes and named color styles to store color sight impaired versions of the colors.
brewsearch finds RGB values of named color styles.
brewcolordb builds a local database of named color styles and uses brewtransform which is a wrapper around libbrewscheme functions to generate additional styles with color sight impaired simulated values.
Yes it really took that long to translate the s2color palette
While there was already a solution in place to recycle colors to the maximum number of colors used in the scheme, it took quite a bit longer for me to do the same with marker symbols.

Now some of those other related issues start to be dealt with.
brewcontrastratio provides a way for users to test the color contrast ratios between background color and text color.
When there is insufficient font size/type face and color contrast ratio it can be extraordinarily difficult for end users to read the text.
If anyone needs to see an example of this, they can look at some of the web pages I've built in the past.
Do you remember the thing about palettes from University of Oregon a few slides ago? brewmorepalettes helps to avoid some of the challenges with including .do files in packages (e.g., required vs optional scripts).
dialog boxes are hard....very hard....but the suggestion from Michael Mitchell was valueable and useful nonetheless.

Lessons learned...
and others I still have yet to learn

Dialog box programming is not easy.

Invest more time/effort into the design of systems/toolkits at the start of a project.

Use version control and make time to review commits over time.

Be aware of dependency management issues and challenges.

Don't be afraid to use other languages (e.g., Mata, Java, Python, etc...) when they provide the best solution to your problem.

But wait...

there's more...

There are a few easter eggs already embedded into the slide deck and I'll be adding more over the next few days.

You can activate the easter eggs by double clicking on the items in the timelines earlier in the slide deck. If you find an easter egg, you'll see a Javascript alert to let you know you've found something and then will be redirected to additional related content.