I have figured out the cause of the lag when you select the pattern bucket fill tool- it is because the side bar loads the pattern images for display. The default pattern set, "papers," has 18 images that are all 1k or higher resolution. It seems that its loading all of them, full res, when you activate the tool, which is slow.
I created a new empty pattern group that has 0 pattern images in it- when that is active, selecting the tool is instantaneous because it doesnt need to load any pattern images for the side bar.
It seems like the solution is either to optimize loading pattern images, or do some sort of smart logic where the a full res pattern image only loads when selected for direct use. But for anyone else experiencing this, it seems that the immediate and practical solution is to simply to keep a custom pattern group active with less and smaller resolution patterns, if lag when switching to the tool is a concern.
This is a huge win for my use of the tool- I switch between pattern bucket and normal bucket rapidly, and have the pattern bucket set to clear blending. this lets me swap between adding and subtracting shapes quickly, so the lag was having a big negative impact.